Multi Level Verification for Automated Testing Systems

Multi level Verification is the process of dynamically managing data for the purpose of 1) Immediate test case verification and/or 2) Deferred test cases evaluation. MLV is an integral component of a Flexible Testing Systems methodology and only practical within that context.

Within the FTS methodology, MLV differentiates between test case verification and evaluation in the following manner. Verification is used to denote a dynamic, immediate, procedure for determining the Pass or Fail status of a test case. Evaluation refers to an examination procedure that occurs after a test suite has completed. Examples of each are provided later in this paper.

Issue

In a Rapid Development Application environment where the interfaces (screens / windows) are continually changing and the data is also constantly being revised and modified, how does one go about designing and building an automated testing system ?

From a developers point of view, one might wonder why this is even a problem in the first place, given the premise that application development naturally takes place in a constantly changing development environment, why should it be any different for testing ? If developers can do it, why not testers ?

This is the issue addressed here - how to build a "Flexible Testing System" that can 1) Automate regression testing of business functionality, 2) Support application developers and 3) Providing valuable information during User Acceptance Testing cycles, regardless that the application, data and environment are in a state of constant transition. Selecting an appropriate tool is essential to building a flexible testing system - AutoTester by Software Recording has such capabilities.

Flexible Testing Systems represent a paradigmatic shift in the approach used to automate the testing of application software as well an the underlying methodology. An understanding of the new testing paradigm requires a brief look at the old testing paradigm and its "taken for granted" assumptions. The old testing paradigm will be referred to as the Rigid Testing Paradigm.

Rigid Testing Paradigm - A Brief History

The Rigid Testing Paradigm is built upon three taken for granted assumption, 1) Embedded Test Script Code , 2) Exact Verification and 3) Informal Test Case formats.

Embedded Test Script Code

Embedded test script code is characterized by the mixing, or embedding, of test case logic, navigation sequences and data within the same test script. For example, the logic to verify the correct window/screen is in focus and the tabbing sequence to access an edit control field on the screen, is combined with the data to be entered into the targeted edit control field. This combination of ingredients produces rigidity resulting in massive test case failure when window verification, navigation, or data, changes due to software bugs or modifications. The rigid, or conventional testing architecture, is not flexible enough to deal with unexpected changes that frequently arise in rapid development environments.

Exact Verification

Exact verification is based on the notion that the evaluation of expected results is either true of false. On the surface this appears to be a sensible idea, however, in practice the "correctness" of results is not always so cut and dry. For one, in a rapid development environment, there may not always be time to refreshed the data prior to each test run. Consequently, results may be somewhat eschewed, but not necessarily incorrect. As it turns out, there are degrees of test results correctness that a more flexible testing system is able to negotiate. Exact verification also assumes that the only worthwhile information a testing system can provide is of the pass/fail nature. Flexible testing system address varying levels of verification and provide a wide range of system and business information helpful to both developers and business users.

Informal Test Case Format

Informal test case format refers to the narrative style characteristic of how test cases are often written. They are informal in the sense that they do not have a well defined format that organizes each test case around specific test criteria. For example, the test case may clearly indicate that the name "John Smith" is expect to appear in the field labeled "Spouse", however, the navigation order or state of the system is assumed rather than specified. In this case, even though expected results are indicated, a systematic format for achieving those results is either lacking or very informal at best. Flexible testing systems replace narratives with specific test case categories that clearly communicate what is being tested.

The obvious question here is "How is it that the Rigid Testing Paradigm was developed, or came into being, in the first place ?" It came into being primarily as a function of 1) Low tech testing tools and 2) Sequential development processes and methodologies, that were available at the time.

Low Tech Testing Tools

Low tech testing tools are limited to "Capture Playback" due to their inability to easily perform field, or object, level verification. Capture playback verification compares a saved screen image to a dynamically captured snapshot of the same screen. Mismatches denote failure, which is often the case if the screens or data changes the least bit. The functionality, or technology, of the testing tools has the effect of shaping and limiting the testing methodology that one can actually apply. The rigid structure imposed by technical limitations has a "domino" effect not only on testing practices but also on the information that testing can provide developer and users.

Sequential Development Methodologies

Sequential development methodologies also contributed to sustaining a Rigid Testing Paradigm, even though its was not their intent to influence how testing was conducted. That is, SDLC's of the time were also restricted by technology. Rapid development methodologies, CASE and prototyping tools were not available or mature. Functional testing of the application more naturally came at the end of the development process when the screens and data were completed and stable. Testing tools were 1) Not intelligent or flexible and 2) Not capable of providing value until the system was mostly completed - again at the end of the development cycle.

The evolution of a new testing paradigm was advanced by the development of high tech, intelligent, testing tools. Intelligent, object oriented, testing tools are the foundation of what I have termed "Automated Flexible Testing Systems". The remained of the article will define the principles, requirements and benefits of Automated Flexible Testing Systems - referred to a "FTS".

Flexible Testing Systems - a brief history

It has been said that "Necessity is the mother of invention". Flexible testing systems were born out of necessity - out of the need to test business applications that were being rapidly developed and constantly changed.

I work in the Information Service Department for Charles Schwab & Co., one of the fastest growing discount brokerage firms in the country. Within ISD, I am a member of the Business Systems Development Testing Group, where we provide testing support for application developers and business users. The development environment is quickly evolving from a centralized "Legacy", CICS mainframe configuration, to a windows based, multi layered, distributed architecture employing CASE tools to rapidly prototype and develop new business applications.

Automated testing systems designed from the Legacy point of view, were no longer able to keep pace in the new RAD environment. They could not be modified quickly enough to stay current with all the application changes. We had to come up with a better way to do our "testing business", if we were going to continue to provide a high quality service to our customers, i.e. application developers and business users.

About two years ago, the Order Entry system was being designed and constructed in the new development environment, while the automated testing system was built the "old fashion" way. It quickly became apparent that the testing system was not working - it could not keep up. Test cases primarily failed because they were unable to correctly locate fields. It got to the point that we had to find a way to deal with the "elusive" field position issue, or get out of the automated testing business. The solution was to create intelligent field navigation and location routines that utilized "relative context awareness". The important understanding here is that relative context awareness allows the test case to vary its execution based on the context at the time of playback instead of at the time of development. Eliminating the field position problem enabled us to run hundreds of test cases, per hour, everyday. This was the first big step toward creating a truly Flexible Testing System.

FTS methodology grew quickly, resolving one issues after another. The "Bad Data" issue turned into the Dynamic Multi Level Verification solution that resulted in expanding our perspective on "what it means" to automate testing in the first place. Gradually transforming FTS into a way of designing and building automated testing system that overcame the challenges of the new CASE, Rapid Application Development Environment.

Flexible Testing Systems - the solution

The solution of the initially stated problem, is to be found in the three foundation of a Flexible Testing System, namely, 1) Structured Test Script Code, 2) Multi-level Verification and 3) Dynamic Test Case Format.

Structured Test Script Code

Structured Test Script Code denote the separation of 1) Test Case Logic, 2) Screen Navigation and 3) Data. This works in the following manner:

A single Screen Program is written to service each screen/window in the system. Screen programs exist independently of test cases, and have three major functions. One, defining the screen itself, the title or label, two, creating variables for all the objects/fields on the screen and three, navigation. The screen program has the intelligence to verify when its screen is up, in focus, and how to locate and determine the status of any object on the screen. Screen programs are CALLed by test cases whenever they needs to service a screen, i.e., entering or validating data.

Navigation

The navigation component of a screen program is central to FTS methodology. It is precisely the ability to find screen objects, that have unexpectedly changed position, that makes a testing system flexible. For example, in version 1.0 of the software, the customer name field is the fourth field on the screen and can be located by pressing the tab key four times. In version 2.0, the customer name field is now the sixth field on the screen and no one has communicated this change to the testers. The question becomes, how to design a testing system so that this unexpected changed does not interrupt the flow of testing ?

FTS methodology handles the above issue in the following manner. Locate the cursor in a "home" position on the screen. This is usually the first field/object located at the top, leftmost, corner of the screen. Capture the cursor's row and column coordinates at home position and store them in temporary variables, (the testing tool should be able to automatically capture row/col coordinates). Move the cursor to the next field/object, with the tab or other navigation key, check to see if its on the correct field, if not, check current row/col position against home position. The idea here is to continue navigating the screen until the cursor is on the correct field or back to its home position.

If the cursor returns to home position without locating the field - then it probably has been removed from the screen or had a drastic name change. At this point, cancel the current test case and begin the next one.

This FTS navigation process will locate fields that have changed position unexpectedly. FTS also incorporate the concept of "Traverse and Mark". This process follows the above navigation description with the advantage of marking where the field should be, reporting it was found in that position or not, and continues to traverse the screen, again marking where it does find the field independent of expectations.

FTS Navigation Code using Cursor Relative Look function

(AutoTester Code by Steve Vance [email protected])

Screen Programs

Another benefit of building screen programs is one point maintenance. That is, when an application screen changes, updating the single corresponding screen program is all that is required. (Remember, test cases are independent of screen programs, so they are not affected.) For example, If a thousand test case have been built that access the Customer Entry Screen, when it changes, instead of having to modify a thousand test cases, all one needs to do is modify the screen program that maps the Customer Entry Screen - a single point of maintenance and the test cases are up and running again. (No harm no foul !)

Data

Data, for individual or logically grouped sets of test cases, is stored in separate ASCII text file. Each test case, defining its own data requirements, will READ an ASCII file containing data to either input or verify.

Two major benefits of storing data in separate ASCII text files are 1) Maintenance and 2) Ownership. Again, single pointed maintenance may be achieved by storing data for a large group of test cases in a single ASCII text file. Additional data can quickly be added or modified to accommodate rapid changes in the system under test. The ease of maintenance refers to the level of technical knowledge require to make data changes. Individual's without technical knowledge of the testing tool can make these changes for all that is required is application knowledge and how to use a word processor or text editor, a very common skill. Users can now maintain their own testing system - even add new test cases, in most instances. This takes the burden off the developers, or testing group, and gives the Users more control and ownership of their Acceptance Testing System. User ownership greatly increased the productivity of the testing system over time, and has far reaching, beneficial effects, for the overall organization.

Given the independence of test data and screen navigation, what remains in an individual test case script? Test case scripts contain 1) READ statement, or loop, to pick up its data requirements, 2) CALL's to Screen programs and logs, in the order specified by the test case, and 3) Information or logic specific to the individual test case, i.e., flags that point to fields/objects required to execute particular business functions for this test case.

Multi Level Verification

Multi level data verification increase the flexibility of the testing system. The more levels of data verification, or evaluation, possible, the more flexible the testing system. Multi level verification, evaluation, is the ability of the testing system to 1) Perform dynamic data verification, at multiple levels and 2) Provide necessary information, system or edit messages, so that an other can evaluate and/or verify the data, at some later point in time, i.e. make determination of pass or fail.

Dynamic data verification is the process whereby the automated testing tool, in real time, GET's data from a field, COMPAREs it to an expected value and writes the result to a log file. It also expresses the ability of the testing tool to make branching decisions based on the outcome of the comparison. Ideally, the testing tool has the ability to combine the GET and COMPARE into one command, known as a LOOK function, to simplify coding dynamic verifications.

Note: Verification refers to correctness of data, while evaluation refers to system information such as system messages or edits. Data is entered into or retrieved from the application. System messages or edits are only displayed by the system as a consequence of a previous action. This is an important distinction from a testing point of view.

In a FTS, dynamic data verifications, or evaluations, can be conducted at seven different levels with two compare modes, inclusive and exclusive, for levels 2 and 3.

The levels are as follows:

Verification

Level 1. Exact field level verification, (1-1)

Level 2. Line Look, inclusive or exclusive

Level 3. Area Looks, inclusive or exclusive

Level 4. Cursor Relative Look

Evaluation

Level 5. Exact Messaging with capture only on error

Level 6. Partial message match with capture

Level 7. Capture message only

The following is a brief description of 4 Verification and 3 Evaluation levels:

VERIFICATION LEVELS

Level 1 Exact Field Level Verification

Level 1 is the most direct and obvious type of dynamic data verification. It does a GET/COMPARE, or LOOK, of a single field and performs a straight forward, one to one, true/false, data evaluation based on an expected result.

Level 2 Line Look, Inclusive or Exclusive

Level 2 does a GET/COMPARE, (or LOOK), for an entire line on the screen. That is, all the text, data, that exists within the boundaries of a pre-define row, is retrieved from the screen and stored in a variable. The COMPARE is set to either inclusive or exclusive. Exclusive means the compare is true if the expected value exist anywhere on the line, and inclusive means the compare is true if the line contains only the expected value.

Level 3 Area Look, Inclusive or Exclusive

Level 3 LOOK's at a pre-defined area of the screen, several rows, or a block, with the compare set to either inclusive or exclusive

Level 4 Cursor Relative Look

Level 4 is a Cursor Relative Look, one of the most important FTS features. This is the ability to LOOK, verify data, relative to the position of the cursor, independent of absolute field row/column coordinates. Cursor relative functionality significantly increases the testing system's flexibility by being able to find, or navigate to, a specific field, even if there has been a change in the field's screen position. This is accomplished by measuring the position of the cursor "relative" to the location of the field. It does not care where a field happens to be at any given moment in time. It only cares where the cursor is in relation to the field.

For example, the Customer Name field is labeled, "Cust Name". Upon entering the field, the cursor is 3 bytes from the label "Cust Name. The Cursor Relative function LOOKs to see that the cursor is in fact 3 bytes from the label "Cust Name", independent of its row/column coordinates. If so, there is a match, the cursor is correctly positioned, and the test cases continues from that point.

Benefits of Cursor Relative Look are 1) Increases a testing systems flexibility and 2) Find fields that are constantly changing position and 3) Reduces maintenance time and effort. The testing system does not have to be updated every time a field changes position - a very big time saver !

(Note: Many testing tools find Cursor Relative Look's difficult to implement. Autotester by Software Recording has built in cursor relative functionality. FTS tool features are discussed in the Tool Requirement Section.)

EVALUATION LEVELS (Messages)

Level 5. Exact Messaging with capture if unexpected

Level 5 looks for an exact system or application message displayed in the message area. In this case the message is capture to a log file only if it is unexpected.

Level 6. Partial message match with capture

Level 6 does not capture/log the message it any part of it is found.

Level 7. Capture message only

Level 7 is the most commonly used evaluation type. All message, edits, are capture/logged whenever they are displayed. This provides the most information for evaluating and debugging a system's performance. The log carefully indicated exactly when and where the message occurs, i.e., screen name, function key, test operation, date and time - along with the message. This level of information makes it easy for developers to identify bugs that may be masked over, even when the expected results, data, matches correctly.

Dynamic Test Case Format

Dynamic Test Case Format clearly represents and displays the dynamic nature of test cases and their major components. Action, Initial State or Screen, Data and Expected Result(s).

Example: #1

Test Case ID: CUST.01

Function: Add a new Customer

Data Assumptions: Customer database has been restored

General Description:

Add a new customer, via the Customer Add screen, and validate that new Customer was displayed corrected on the All Customer Screen.

1. Easier to automated - all have same structure.

2. DATA requirements are clearly defined.

3. Navigation is precise

4. Means to Expected results is specific - no guess work involved.

5. Increased likelihood of correctly testing the function - not find out later testing the wrong things

Tool Requirements for a Flexible Test System

In order to support the development of a flexible, automated test system, the test tool must have four basic characteristics:

1) Fixed and relative context awareness

2) Variables

3) Variable indirection

4) Logic and branching

Each of these characteristics and its impact on a flexible test system is discussed in more detail below.

1. Fixed and relative context awareness

Context awareness refers to the ability of the tool to recognize where it is in the application or system under test. In most GUI environments, this includes such features as recognizing which window is active and which control has focus; in character-based systems, this also includes recognizing the location of the cursor or defined text strings. Context awareness is important because it allows the test to vary its execution based on the context at the time of playback instead of at the time of development, thus minimizing the impact of changes in the application on test playback.

The concept of "relative" context awareness means that the context of one object or control can be determined with reference to the position or location of another. In a GUI environment, where windows can be sized and repositioned, relative context means both the relative position of an item within a window as well as its aspect ratio, or relationship to the size of the window. For text-based applications, this means being aware of the location of a text string, for example, relative to the position of the cursor. This relativity of context awareness is critical for reducing or minimizing the effect of otherwise cosmetic changes, such as moving a window or a field.

2. Variables

Variables are named elements or objects in the test script whose content can be varied during execution. An obvious type of variable is an edit control in a GUI application or an input field in a text system; a perhaps less obvious type is a window or screen. By assigning a variable to contain the actual string of input characters, the contents of the control or field can be varied from one test case or run to another without duplicating the entire test script for each iteration. The potential values for a variable can then be supplied either from a external file, from the user at the keyboard, as a result of calculations, or from another window or screen in the application.

For example, a series of transactions may be created in a spreadsheet or extracted from a database into an exchange file format, such as ASCII. Then, by defining a loop in the test tool that reads each record consecutively, substituting the file values for each defined variable in the test script, a test sequence that processes a single transaction can be transformed into one that processes an unlimited number of transactions. These external values can supply not only inputs but also expected outputs as well, including responses such as error messages or state transitions.

3. Variable Indirection

Variable indirection is one additional dimension of variability. In this case, a variable is used to store the name of the target variable: the term indirection comes from the fact that the first variable "points to" to the second, so that it is accessed indirectly. This level of abstraction greatly compresses the number and size of test scripts by requiring only one script per common element or control within a system.

For example, a test script could be created to verify standard requirements of a certain type of control, such as a check box: setting the focus with the keyboard, setting the focus with the mouse, setting the focus with a mnemonic, verifying the status of the check box, and setting the status to either checked or unchecked. By using a variable to name each control and its parameters, then using another variable to indicate which control is active, a single set of tasks can be used to verify the behavior of any check box in the system by simply passing the name of the control and its parameters to the test script.

Variable Indirection - an example

As an example of using variable indirection in test case development, imagine a window named "Update Employee" with an Edit Control name "Last Name".

In order to verify that the field contains the value "Smith", the text-file-based test script has the following commands:

1. Verify Panel, Update Employee

2. Verify Field, Last Name, "Smith"

The window variable to support the above verification are:

UPDATE_EMPLOYEE.MAIN (The panel itself)

UPDATE_EMPLOYEE.LAST_NAME (The edit control field)

the code for the test script is:

1. Verify Panel, UPDATE_EMPLOYEE

2. Verify Field, LAST_NAME, "Smith"

The testing tool would then concatenate the panel name, "UPDATE_EMPLOYEE", and the field name, "LAST_NAME, (the name of the window variable) as a text string in a temporary text variable (e.g. TEMP.TEXT). Then by the using variable indirection (variable TEMP.TEXT contains the name of another variable, UPDATE_EMPLOYEE.LAST_NAME, the value "Smith" could be verified directly.

This variable indirection feature allows the automated test system designer to write a small number of highly capable control programs that, along with the careful use of English-like variable naming standards, provides a self-documenting, easy to read, testing system.

(Variable indirection example care of Marin Data System)

4. Logic and branching

Logic refers to the ability of the tool to make decisions based on conditions at the time of playback, and vary the sequence of test execution as a result. The most basic level includes if/then/else

decisions that execute or skip instructions within a single test script based on the outcome of the decision. For example, the test script might first verify whether a particular control or field has focus before input is attempted; if the focus is not correct, the test script could set the focus before proceeding.

Another level of logic and branching includes the capability to branch from one test script to another, then return to the original one and continue. This is known as "nesting", because one or more test scripts can be nested, or included within, another. For example, a test script that encounters an error during playback might branch to another script whose function is to log the error, then return to the original script and continue execution. However, the error logging script might also verify whether the error is a problem with context, and if so it is required to restore the expected context before playback continues. To accomplish this, it might also branch to a test script whose function is to recover the context or state of the application to a known place. The error logging script is nested within the test script, and the context recovery script is nested within the error logging script.

The advantage of logic and branching is that it supports modularity and flexibility. Modularity means that common tasks, such as logging errors or recovering context, can be developed just once but shared across all other scripts. This not only saves development time, but permits single point maintenance when a change is needed to a commonly used task. If all scripts use the same error logging routine, for example, a decision to capture new system status information as part of documenting an error would need to be made in only one place, instead of in every script that could potentially encounter an error.

Flexibility is also maximized by logic and branching, as it allows a single test script to modify its behavior based on results at playback. Otherwise, a separate test script would be required for each possible condition or pathway, or the exact state of the system would have to be known beforehand. For example, if the test script needed to create a new file of a certain name, it could first verify that there was no existing file of the same name; if one was found, the steps to delete it could be executed first. This flexibility means the state of test data and files would not have to be as rigidly controlled in order to maximize the probability that the test would execute successfully.

Note: AutoTester by Software Recording meets or exceeds all the FTS tool requirements.

Conclusions

Flexible Testing System is an evolving technology aimed at increasing longevity and decreasing maintenance.

Longevity reflects the automated testing systems ability to intelligently adjust and respond to unexpected changes to the application under test. By competently doing so, the testing system's life expectancy naturally increases through its inherent ability to stay useful overtime.

Reduced maintenance implies not only time but necessity as well. Time savings is a function of the elimination of redundancy, exemplified by features such as "One Point" maintenance. Necessity reduction, for example, is the result of intelligent design features characterized by "Heuristic Navigation".

FTS evolution begins to provide solutions to testing challenges arising from new development technologies and environments. It is important to remember that flexible testing systems execute test cases, they do not define or create them. Therefore, FTS is only a piece of the automation puzzle and not a final solution. FTS principles may one day be applied to a model for automating the definition and creation of test cases. Moving closer to goal of realizing a system that automatically test itself !